System, method and computer program product for remote graphics processing

ABSTRACT

A system, method, and computer program product are provided for remote rendering of computer graphics. The system includes a graphics application program resident at a remote server. The graphics application is invoked by a user or process located at a client. The invoked graphics application proceeds to issue graphics instructions. The graphics instructions are received by a remote rendering control system. Given that the client and server differ with respect to graphics context and image processing capability, the remote rendering control system modifies the graphics instructions in order to accommodate these differences. The modified graphics instructions are sent to graphics rendering resources, which produce one or more rendered images. Data representing the rendered images is written to one or more frame buffers. The remote rendering control system then reads this image data from the frame buffers. The image data is transmitted to the client for display or processing. In an embodiment of the system, the image data is compressed before being transmitted to the client. In such an embodiment, the steps of rendering, compression, and transmission can be performed asynchronously in a pipelined manner.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation and claims the prioritybenefit of U.S. patent application Ser. No. 14/987,690 filed Jan. 4,2016, which is a continuation and claism the priority benefit of U.S.patent application Ser. No. 14/303,566 filed Jun. 12, 2014, now U.S.Pat. No. 9,230,295 that issued Jan. 5, 2016, which is a continuation andclaims the priority benefit of U.S. patent application Ser. No.14/042,659 filed Sep. 30, 2013, now U.S. Pat. No. 8,760,456 issued Jun.24, 2014, which is a continuation and claims the priority benefit ofU.S. patent application Ser. No. 13/850,250 filed Mar. 25, 2013, nowU.S. Pat. No. 8,581,917 issued Nov. 12, 2013, which is a continuationand claims the priority benefit of U.S. patent application Ser. No.11/892,627 filed Aug. 24, 2007, now U.S. Pat. No. 8,427,491 issued Apr.23, 2013, which is a continuation and claims the priority benefit ofU.S. patent application Ser. No. 09/629,458 filed Jul. 31, 2000, nowU.S. Pat. No. 7,274,368 issued Sep. 25, 2007, the disclosures of whichare incorporated herein by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to computer graphics, and moreparticularly to graphics computing architecture.

Description of the Related Art

Today a user of a graphics processing host, such as an INFINITE REALITYsystem by Silicon Graphics, Inc. of Mountain View, Calif., runs agraphics application locally. Rendering in response to the graphicsinstructions is also performed locally. The distance from the host to auser can be extended to a distance of, for example, one to threekilometers by the use of fiberoptic cables to connect the user's monitorand keyboard with the graphics processing host. More important, the hostis the dedicated resource of a single person. It can only be shared ifsomeone else takes the place of the current user. While a dedicatedresource is convenient for the user, it may not be economical. The costof a graphics processing host is considerable; less than full usage ofsuch a host lowers its cost-effectiveness. This lower cost-effectivenesscould deter an organization from procuring a graphics processing host.

Hence there is a need for a system and method by which additional userscan utilize a remote graphics processing host. This would improve thecost-effectiveness of the host. Specifically, a system and method areneeded where a user can execute a graphics application at a remotegraphics processing host, causing rendering to be performed at the hostsuch that the resulting images are returned to the user.

SUMMARY OF THE PRESENTLY CLAIMED INVENTION

The invention described herein is a system, method, and computer programproduct for remote rendering of computer graphics. The inventionincludes a graphics application program resident at a remote server. Thegraphics application is invoked by a user or process located at aclient. At the server, the invoked graphics application proceeds toissue graphics instructions. The graphics instructions are received by aremote rendering control system. Given that the client and server differwith respect to graphics context and image processing capability, theremote rendering control system modifies the graphics instructions inorder to accommodate these differences. The modified graphicsinstructions are sent to graphics rendering resources, which produce oneor more rendered images. Data representing the rendered images iswritten to one or more frame buffers. The remote rendering controlsystem then reads this image data from the frame buffers. The image datais transmitted to the client for display or processing. In an embodimentof the invention, the image data is compressed before being transmittedto the client. In such an embodiment, the steps of rendering,compression, and transmission can be performed asynchronously in apipelined manner.

Features and Advantages

The invention described herein has the feature of allowing a user toinvoke a graphics application at a remote server. The invention also hasthe feature of allowing the graphics application to send graphicsinstructions to graphics rendering resources located with the server, sothat rendering is performed at the host and not at the user's machine.The invention also has the feature of returning rendered image data tothe user.

The invention has the advantage of allowing a user to utilize graphicsresources that are not co-located with the user, but are instead locatedat a distance from the user. The invention also has the advantage ofallowing greater utilization of graphics resources, since users otherthan local users can take advantage of graphics resources.

The foregoing and other features and advantages of the invention will beapparent from the following, more particular description of a preferredembodiment of the invention, as illustrated in the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the overall architecture of an embodiment of theinvention.

FIG. 2 is a block diagram of the remote rendering control system,according to an embodiment of the invention.

FIG. 3 is a flowchart illustrating the method of the invention,according to an embodiment thereof.

FIG. 4 is a flowchart illustrating the step of initialization, accordingto an embodiment of the invention.

FIG. 5 is a flowchart illustrating the client/server handshake,according to an embodiment of the invention.

FIG. 6 is a flowchart illustrating the graphics application=s receipt ofa client window, according to an embodiment of the invention.

FIG. 7 is a flowchart illustrating the graphics application=s receipt ofa graphics context, according to an embodiment of the invention.

FIG. 8 is a flowchart illustrating the binding of a server context to aserver window, according to an embodiment of the invention.

FIG. 9 is a flowchart illustrating the step of imposing clientparameters on the rendering process, according to an embodiment of theinvention.

FIG. 10 illustrates the pipelining of the rendering, compression, andtransmission operations, according to an embodiment of the invention.

FIG. 11 is a flowchart illustrating the remote rendering process fromthe perspective of the client, according to an embodiment of theinvention.

FIG. 12 illustrates the computing environment of the invention,according to an embodiment thereof.

DETAILED DESCRIPTION

AA preferred embodiment of the present invention is now described withreference to the figures where like reference numbers indicate identicalor functionally similar elements. Also in the figures, the left mostdigit of each reference number corresponds to the figure in which thereference number is first used. While specific configurations andarrangements are discussed, it should be understood that this is donefor illustrative purposes only. A person skilled in the relevant artwill recognize that other configurations and arrangements can be usedwithout departing from the spirit and scope of the invention. It will beapparent to a person skilled in the relevant art that this invention canalso be employed in a variety of other systems and applications.

Contents

I. Indroduction II. Apparatus III. Method

A. Server Processing

1. Initialization

a. Client/Server Handshake

b. Client Window

c. Graphics Context

d. Context and Server Window

2. Client Parameters

3. Rendering and Transmission

B. Client Processing

IV. Environment V. Conclusion I. Introduction

The invention described herein is a system, method, and computer programproduct for remote rendering of computer graphics. The invention allowsa user or process at a client computer to access remotely locatedrendering resources, such as a graphics processing host embodied in orco-located with a remotely located server. According to the invention, agraphics application executing at the server sends graphics instructionsto a remote rendering control system. The remote rendering controlsystem can be embodied in software running on the server. Theinstructions sent from the application are not sent to the client. Hencethe client does no rendering. The instructions are modified by theremote rendering control system and sent to graphics renderingresources. Rendering consequently takes place at the graphics resources.The image data that is produced by the graphics resources are then sentto the client.

II. Apparatus

The basic architecture of an embodiment of the present invention isillustrated in FIG. 1. A user or process at a client machine 103 wishesto have rendering performed remotely. Accordingly, client 103 issuescommands 107 to a remotely located server 109. In an embodiment of theinvention, connectivity between client 103 and server 109 is establishedand maintained through a communications network 115, such as theInternet. Commands 107 are received at server 109 by a graphicsapplication 120. In response to commands 107, application 120 generatesgraphics instructions 125. Graphics instructions may, for example, be inthe OPENGL language.

Graphics instructions 125 are sent to a remote rendering control system130. Remote rendering control system 130 accepts graphics instructions125 and modifies them to create modified graphics instructions 135. Anexample of a remote rendering control system is the VIZSERVER softwaresystem produced by Silicon Graphics, Inc. As will described in greaterdetail below, the modifications to graphics instructions 125 areperformed to take into account the differences between server 109 andclient 103 with respect to their imaging processing capabilities andgraphics contexts. In response to modified graphics instructions 135,graphics resources 140 render one or more images and return image data145 to remote rendering control system 130. In an embodiment of theinvention, image data 145 is then compressed to form compressed imagedata 150. Compressed image data 150 is then sent to client 103, vianetwork 115. In alternative embodiments of the invention, compression isnot implemented.

A more detailed logical illustration of remote rendering control system130 is shown in FIG. 2. Remote rendering control system 130 acceptsgraphics instructions 125 through an interface 205 which is transparentto the graphics application 120. Graphics application 120 is effectivelysending graphics instructions to local rendering resources instead of toclient 103. Because of the transparency of interface 205, graphicsapplication 120 need not be aware of this redirection. Transparentinterface 205 performs a variety of modifications to graphicsinstructions 125, to produce modified graphics instructions 135. Themodifications performed by transparent interface 205 will be describedin greater detail below. Modified graphics instructions 135 are thensent to graphics resources 140 (not shown in FIG. 2) to effectrendering.

After rendering has been performed, remote rendering control system 130accepts the resulting image data 145. As described above, in anembodiment of the invention, image data 145 undergoes compression beforetransmission to client 103. Compression of the data is performed by adata compression module 210, which produces compressed image data 150.Various methods of data compression are known to persons of ordinaryskill in the art. In an embodiment of the invention, color cellcompression is used. In alternative embodiments of the invention, datacompression module 210 can perform alternative compression algorithms,such as the process described in U.S. patent application Ser. No.09/458,011, now U.S. Pat. No. 7,274,368, “Image Data Compression andDecompression,” incorporated herein by reference in its entirety. In anembodiment of the invention, data compression module 210 is capable ofperforming any of a plurality of data compression algorithms.

III. Method

The overall process of an embodiment of the invention is illustrated inFIG. 3. Process 300 starts at step 305. At step 310, a graphics sessionbetween a client and a remote server is initialized. Initialization step310 includes the clients initial access to and protocol “handshake” withthe server. In addition, step 310 includes the starting of the graphicsapplication. Step 310 also includes reconciling the differences betweenthe client and server with respect to graphics processing capability andgraphics context. Such reconciliation is necessary to allow processingof an image at the server and display (or further processing) of theimage at the client. Initialization will be described in greater detailbelow.

In step 320, the graphics application generates graphics instructionsfor the remote rendering control system. In step 325, parametersspecific to the client are imposed. This step includes, for example,interception by the transparent interface of function calls andreferences that occur in the graphics instructions, and the modificationof those function calls and references to make them suitable to theclient. In particular, adjustments may have to be made given that theclient and server can have different graphics processing capabilitiesand contexts. Step 325 is described in greater detail below.

In step 330, the remote rendering control system sends modified graphicsinstructions to the graphics resources. The modified instructions ofthis step correspond to the graphics instructions generated in step 320and modified in step 325. In step 335, the graphics resources render oneor more images according to the modified instructions of step 330.

In step 340, after rendering is completed, the remote rendering controlsystem reads the resulting image data from a frame buffer. In anembodiment of the invention, the graphics application can have one ormore specific instructions that force the reading of the frame buffer.If, for example, OPENGL is being used, the instruction “glflush” causesthe frame buffer to be “flushed,” i.e., read. The OPENGL instructionglswapbuffers” also serves this purpose where two frame buffers areused. Here, the buffers will be swapped. A first buffer, to which imagedata has most recently been written, will be read. Writing, meanwhile,now takes place into the second buffer. Likewise, at the next“glswapbuffers” instruction, the second buffer will be read and thefirst buffer will begin receiving new image data.

In step 345, the image data is enqueued for purposes of compression. Instep 350, the image data is compressed so that in step 355 the imagedata can be transmitted to the client efficiently. In step 360, adetermination is made as to whether additional rendering is necessary.If so, the process returns to step 320. If no additional rendering isrequired in step 360, the process concludes at step 365.

A. Server Processing

1. Initialization

The initialization step 310, according to an embodiment of theinvention, is illustrated in greater detail in FIG. 4. The processbegins at step 405. In step 410, a user or process at the clientmachine, begins logging on to the server. As a result, in step 415 aclient/server handshake takes place. The handshake includes the initialcontact and protocol establishment between the server and client. Step415 also includes an agreement process between the client and server asto which data compression algorithm will be used, if there are more thanone available. The client/server handshake process is described ingreater detail below with respect to FIG. 5. In step 420, the clientexports to the server a list of the client's graphics capabilities. Alist of specific graphics capabilities is known as a visual. Examples ofcapabilities that may be included in a visual are the ability to performstenciling or represent opacity. Another component of a visual is thecolor mapping range of a device, e.g., 48-bit versus 24-bit colorrepresentation.

In step 425, the user or process starts a console window at the clientcomputer. In step 430, the user or process starts the graphicsapplication resident at the server. In step 435, the transparentinterface to the graphics application opens the client display and theserver display. In step 440, the transparent interface merges the visualof the client with the visual of the server. Because a graphicsapplication typically deals with a single output device, the graphicsapplication must see a single visual. This merger allows the graphicsapplication to see a single visual, while in reality presenting visualsfor both client and server. In step 445, the transparent interfaceassociates the client display with the graphics application. In step450, the transparent interface overlays the server visual with a routinethat allows conversion of visual capabilities appearing in functioncalls of the graphics application. The routine converts these visualcapabilities to capabilities appropriate to the client, i.e.,capabilities in the client's original visual. The development andapplication of such a routine would be obvious to one of ordinary skillin the art given this description.

In step 455, the client window is returned to the graphics application.This step includes the conversion, by the transparent interface, of themerged visual list into a visual appropriate to the client. Step 455also includes creation of an internal data structure for tracking theposition of the client window in the client display. The process ofreceiving the client window is described in greater detail below withrespect to FIG. 6.

In step 460, the application receives a graphics context. This step,detailed below with respect to FIG. 7, includes definition of a servercontext by the transparent interface. Step 460 also includes the returnof an internal context to the application. In step 465, the applicationbinds the internal context to the client window, which has the effect ofbinding the server context to the server window. The initializationprocess concludes with step 470.

a. Client/Server Handshake

An embodiment of the client/server handshake, step 415, is illustratedin greater detail in FIG. 5. This process begins with step 505. In step510, the server connects to the client. In step 515, the server (inparticular, the application) identifies an address or source from whichupdates of image data are provided. It is from this source that updateswill be sent to the client. In step 520, the server and client identifytheir respective available compression algorithms to each other. In step525, the client chooses a compression algorithm and identifies it to theserver. The process concludes with step 530.

b. Client Window

The step by which a client window is returned to the application, step455, is illustrated in greater detail in FIG. 6, according to anembodiment of the invention. The process begins with step 605. In step610, the transparent interface produces a visual appropriate for theclient from the merged visual list. This is done by taking only thosevisual capabilities which were originally the client's. Definition ofsuch a visual is required for an image to be displayed properly at theclient. In step 615, the transparent interface defines a window for theclient. In step 620, the transparent interface creates an internal datastructure for tracking the position of the client window on the clientdisplay. In step 625, the transparent interface returns the clientwindow to the application. The process concludes with step 630.

c. Graphics Context

An embodiment of the process by which a graphics context is returned tothe application, step 460, is illustrated in greater detail in FIG. 7.The process begins with step 705. In step 710, the transparent interfaceproduces a server visual from the merged visual. This is done by takingonly those visual capabilities which were originally the server's. Instep 715, the transparent interface defines a server context. The servercontext represents the context settings to be applied in rendering forthe server's display. In step 720, the transparent interface returns aninternal context to the application program, where the internal contextincludes the server context and any additional information that may benecessary to allow the transparent interface to properly interpretreferences to the server context. An example of such additionalinformation is the identity of specific visual capabilities thatcorrespond to a given element of the server context. The processconcludes with step 725.

d. Context and Server Window

A The step of binding a context to a server window, step 465, isillustrated in greater detail in FIG. 8, according to an embodiment ofthe invention. The process begins with step 805. In step 810, thetransparent interface extracts the server context from the internalcontext. In step 815, the transparent interface requests a windowallocation from the session manager. The session manager is logic (e.g.,software) responsible for managing the allocation of resources in agraphics session. The implementation of a session manager is known topersons of ordinary skill in the art. In step 820, the transparentinterface binds the server context to the server window. The processconcludes with step 825.

2. Client Parameters

Returning to the overall process 300 illustrated in FIG. 3, onceinitialization is completed in step 310, the graphics applicationgenerates graphics instructions in step 320, as discussed above. This isfollowed by step 325, the step of imposing client parameters on therendering process. An embodiment of step 325 is illustrated in greaterdetail in FIG. 9. The process begins with step 905. In step 910, thetransparent interface intercepts graphics instruction function callsthat include a visual. In step 915, the transparent interface convertsthe visuals to corresponding client visuals. In step 920, thetransparent interface intercepts every graphics instruction reference toa context. In step 925 the transparent interface converts the referenceto a reference consistent with the client's context. The processconcludes at step 930.

3. Rendering and Transmission

Steps 340 through 355 (rendering, enqueuing, compression, andtransmission) are collectively illustrated in FIG. 10, according to anembodiment of the invention. This figure illustrates how blocks of imagedata are rendered, compressed, then transmitted. In the illustratedembodiment, these operations are performed in pipeline fashion. Therendering, compression, and transmission steps are identifiedcollectively as processes 1005. These steps take place sequentiallyduring successive blocks of time 1010. During time t.sub.0, a block A ofimage data is rendered. During the next interval, time t.sub.1, block Ais compressed. Simultaneously, the next segment of image data, block B,is rendered. At the next time interval, t.sub.2, block A is transmittedafter having been compressed during the previous time period. Whileblock A is being transmitted, block B is compressed. Simultaneously, thenext block of image data, block C, is being rendered. Successive blocksof image data are processed in this manner.

In an embodiment of the invention, the steps of rendering, compression,and transmission are asynchronous. In such an embodiment, thecompression of block B may not be completed at the same time as therendering of block C, referring to the example above. To deal with this,blocks of image data may be discarded. Any given step will take only themost recent block from the previous step. If, for example, rendering ofblock B takes place faster than the compression of block A, thecompression module is not yet ready for block B when rendering of blockB is done. Block B will therefore be discarded. The compression modulewill finish compressing block A, then wait for the next (and newest)block to emerge from rendering. Multiple blocks may be discarded ifrendering is significantly faster than compression. The transmissionprocess will likewise accept only the newest compressed block of imagedata. If compression of a block is completed before the previous blockhas been transmitted, that compressed block will be discarded.

B. Client Processing

An embodiment of the remote rendering process from the perspective ofthe client is illustrated in FIG. 11. The process starts with step 1105.In step 1110, the client/server handshake takes place, as describedabove. In step 1115, the client sends one or more commands to thegraphics application to begin execution. In step 1120, after thegraphics resources at the server have completed rendering, the clientreceives image data via the network. In an embodiment of the invention,the image data is compressed for purposes of transmission efficiency. Instep 1125, the image data is decompressed at the client and in step1130, the appropriate image is drawn to a window at the client. In step1135 a determination is made as to whether an additional image is to bedrawn. If so, the process continues at step 1120. Otherwise the processconcludes at step 1140.

IV. Environment

Server 109 of the present invention may be implemented as a computersystem or other processing system. Remote rendering control system 130and graphics application program 120 can be implemented as computerprograms executing on such a computer system. Graphics resources 140 canbe any type of graphics subsystem linked to a communicationinfrastructure 1206, such as a bus or network. An example of such acomputer system 1200 is shown in FIG. 12. The computer system 1200includes one or more processors, such as processor 1204. The processor1204 is connected to communication infrastructure 1206. Various softwareimplementations are described in terms of this exemplary computersystem. After reading this description, it will become apparent to aperson skilled in the relevant art how to implement the invention usingother computer systems and/or computer architectures.

Computer system 1200 also includes a main memory 1208, preferably randomaccess memory (RAM), and may also include a secondary memory 1210. Thesecondary memory 1210 may include, for example, a hard disk drive 1212and/or a removable storage drive 1214, representing a floppy disk drive,a magnetic tape drive, an optical disk drive, etc. The removable storagedrive 1214 reads from and/or writes to a removable storage unit 1218 ina well known manner. Removable storage unit 1218, represents a floppydisk, magnetic tape, optical disk, or other storage medium which is readby and written to by removable storage drive 1214. As will beappreciated, the removable storage unit 1218 includes a computer usablestorage medium having stored therein computer software and/or data.

In alternative implementations, secondary memory 1210 may include othermeans for allowing computer programs or other instructions to be loadedinto computer system 1200. Such means may include, for example, aremovable storage unit 1222 and an interface 1220. Examples of suchmeans may include a program cartridge and cartridge interface (such asthat found in video game devices), a removable memory chip (such as anEPROM, or PROM) and associated socket, and other removable storage units1222 and interfaces 1220 which allow software and data to be transferredfrom the removable storage unit 1222 to computer system 1200.

Computer system 1200 may also include a communications interface 1224.Communications interface 1224 allows software and data to be transferredbetween computer system 1200 and external devices. Examples ofcommunications interface 1224 may include a modem, a network interface(such as an Ethernet card), a communications port, a PCMCIA slot andcard, etc. Software and data transferred via communications interface1224 are in the form of signals 1228 which may be electronic,electromagnetic, optical or other signals capable of being received bycommunications interface 1224. Commands 107 and compressed image data150 are examples of signals 1228. These signals 1228 are provided tocommunications interface 1224 via a communications path (i.e., channel)1226. This channel 1226 carries signals 1228 and maybe implemented usingwire or cable, fiber optics, a phone line, a cellular phone link, an RFlink and other communications channels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to media such as removablestorage units 1218 and 1222, a hard disk installed in hard disk drive1212, and signals 1228. These computer program products are means forproviding software to computer system 1200.

Computer programs (also called computer control logic) are stored inmain memory 1208 and/or secondary memory 1210. Computer programs mayalso be received via communications interface 1224. Remote renderingcontrol system 130 and graphics application program 120 can beimplemented as computer programs. Such computer programs, when executed,enable the computer system 1200 to implement the present invention asdiscussed herein. In particular, the computer programs, when executed,enable the processor 1204 to implement the present invention.Accordingly, such computer programs represent controllers of thecomputer system 1200. Where the invention is implemented using software,the software may be stored in a computer program product and loaded intocomputer system 1200 using removable storage drive 1214, hard drive 1212or communications interface 1224.

V. Conclusion

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample, and not limitation. It will be apparent to persons skilled inthe relevant art that various changes in detail can be made thereinwithout departing from the spirit and scope of the invention. Thus thepresent invention should not be limited by any of the above-describedexemplary embodiments, but should be defined only in accordance with thefollowing claims and their equivalents.

What is claimed is:
 1. A method for processing graphics, the methodcomprising: sending information over a network interface to a server,the information identifying a graphics capability of a client; sending agraphics command over a network interface to a server, wherein theserver: merges the graphics capability of the client with one or moregraphics capabilities of the server into a list of graphicscapabilities, receives the graphics command, updates image data at theserver according to the graphics command and according to at least onecapability of the one or more graphics capabilities in the list ofgraphics capabilities, renders client image data for transmission to theclient from the updated image data according to the identified graphicscapability of the client in the list of graphics capabilities, andtransmits the rendered image for display at the client; receiving theclient image data; and displaying the client image data on a display atthe client.